thread-stream

What is thread-stream?

The thread-stream npm package is designed to facilitate the creation of writable streams that operate within worker threads in Node.js. This allows for offloading CPU-intensive tasks or I/O-bound tasks to separate threads, improving the performance and responsiveness of Node.js applications.

What are thread-stream's main functionalities?

Creating a basic thread stream

This code demonstrates how to create a basic thread stream using the thread-stream package. It involves specifying a worker file and optional worker data. The stream created can be used like any other writable stream in Node.js.

const ThreadStream = require('thread-stream');
const stream = new ThreadStream({
  filename: path.join(__dirname, 'worker.js'),
  workerData: { exampleData: 'data' },
  sync: false
});
stream.write('Hello, world!');
stream.end();

Handling stream events

This example shows how to handle data and end events from a thread stream. It allows processing of data received from the worker thread and handling the end of the stream.

stream.on('data', (data) => {
  console.log('Data from stream:', data);
});
stream.on('end', () => {
  console.log('Stream ended.');
});

Other packages similar to thread-stream

workerpool

Workerpool is a package that manages a pool of workers and handles the execution of tasks in these workers. It is similar to thread-stream in that it helps offload tasks to separate threads but differs in its approach by managing multiple workers and tasks rather than focusing on stream-based communication.

bthreads

Bthreads provides a wrapper around Node.js worker threads, simplifying thread management and communication. While it shares the goal of enhancing performance through threads like thread-stream, bthreads does not specifically focus on stream-based interfaces, offering a broader range of thread management features.

README

thread-stream

A streaming way to send data to a Node.js Worker Thread.

install

npm i thread-stream

Usage

'use strict'

const ThreadStream = require('thread-stream')
const { join } = require('path')

const stream = new ThreadStream({
  filename: join(__dirname, 'worker.js'),
  workerData: { dest },
  workerOpts: {}, // Other options to be passed to Worker
  sync: false, // default
})

stream.write('hello')

// Asynchronous flushing
stream.flush(function () {
  stream.write(' ')
  stream.write('world')

  // Synchronous flushing
  stream.flushSync()
  stream.end()
})

In worker.js:

'use strict'

const fs = require('fs')
const { once } = require('events')

async function run (opts) {
  const stream = fs.createWriteStream(opts.dest)
  await once(stream, 'open')
  return stream
}

module.exports = run

Make sure that the stream emits 'close' when the stream completes. This can usually be achieved by passing the autoDestroy: true flag your stream classes.

The underlining worker is automatically closed if the stream is garbage collected.

External modules

You may use this module within compatible external modules, that exports the worker.js interface.

const ThreadStream = require('thread-stream')

const modulePath = require.resolve('pino-elasticsearch')

const stream = new ThreadStream({
  filename: modulePath,
  workerData: { node: 'http://localhost:9200' }
})

stream.write('log to elasticsearch!')
stream.flushSync()
stream.end()

This module works with yarn in PnP (plug'n play) mode too!

Emit events

You can emit events on the ThreadStream from your worker using worker.parentPort.postMessage(). The message (JSON object) must have the following data structure:

parentPort.postMessage({
  code: 'EVENT',
  name: 'eventName',
  args: ['list', 'of', 'args', 123, new Error('Boom')]
})

On your ThreadStream, you can add a listener function for this event name:

const stream = new ThreadStream({
  filename: join(__dirname, 'worker.js'),
  workerData: {},
})
stream.on('eventName', function (a, b, c, n, err) {
  console.log('received:', a, b, c, n, err) // received: list of args 123 Error: Boom
})

Post Messages

You can post messages to the worker by emitting a message event on the ThreadStream.

const stream = new ThreadStream({
  filename: join(__dirname, 'worker.js'),
  workerData: {},
})
stream.emit('message', message)

On your worker, you can listen for this message using worker.parentPort.on('message', cb).

const { parentPort } = require('worker_threads')
parentPort.on('message', function (message) {
  console.log('received:', message)
})

License

MIT